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Using electronic health record data to develop 
inpatient mortality predictive model: Acute 
Laboratory Risk of Mortality Score (ALaRMS) 
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ABSTRACT 

Objective Using numeric laboratory data and 
administrative data from hospital electronic health record 
(EHR) systems, to develop an inpatient mortality 
predictive model. 

Methods Using EHR data of 1 428 824 adult 
discharges from 70 hospitals in 2006-2007, we 
developed the Acute Laboratory Risk of Mortality Score 
(ALaRMS) using age, gender, and initial laboratory 
values on admission as candidate variables. We then 
added administrative variables using the Agency for 
Healthcare Research and Quality (AHRQ)'s clinical 
classification software (CCS) and comorbidity software 
(CS) as disease classification tools. We validated the 
model using 770 523 discharges in 2008. 
Results Mortality predictors with ORs >2.00 included 
age, deranged albumin, arterial pH, bands, blood urea 
nitrogen, oxygen partial pressure, platelets, pro-brain 
natriuretic peptide, troponin I, and white blood cell 
counts. The ALaRMS model c-statistic was 0.87. Adding 
the CCS and CS variables increased the c-statistic to 0.91. 
The relative contributions were 69% (ALaRMS), 25% 
(CCS), and 6% (CS). Furthermore, the integrated 
discrimination improvement statistic demonstrated a 127% 
(95% CI 122% to 133%) overall improvement when 
ALaRMS was added to CCS and CS variables. In contrast, 
only a 22% (CI 19% to 25%) improvement was seen 
when CCS and CS variables were added to ALaRMS. 
Conclusions EHR data can generate clinically plausible 
mortality predictive models with excellent discrimination. 
ALaRMS uses automated laboratory data widely available 
on admission, providing opportunities to aid real-time 
decision support. Models that incorporate laboratory and 
AHRQ's CCS and CS variables have utility for risk 
adjustment in retrospective outcome studies. 



INTRODUCTION 

Mortality predictive models incorporating objective 
clinical data enhance clinical validity. With the 
deployment of electronic health record (EHR) 
systems, clinical data, especially numeric laboratory 
data, are becoming w^idely automated. Rapid assess- 
ment of clinical severity using EHR data available 
at the time of admission may aid decision support 
and improve healthcare quality. Previous studies 
have demonstrated that laboratory data contribute 
most to predicting mortality in hospitalized patients 
in both disease-specific and generic models in the 
US patient population. Studies from other coun- 
tries have also demonstrated the high predictive 
value for laboratory data.'^"^ 

The disease-specific risk adjustment model is devel- 
oped for patients w^ith specific clinical conditions (eg. 



pneumonia, septicemia, heart failure, acute myocar- 
dial infarction).^"^ Although disease-specific models 
are commonly used for outcome studies and hospital 
profiling, their application is limited to high-volume 
clinical conditions, w^hile low^-volume clinical condi- 
tions are less studied. Eurthermore, disease specific 
models usually rely on discharge principal diagnosis 
for disease classification, w^hich limits real-time 
applications because discharge diagnoses are not 
available until after patients are discharged from the 
hospital. 

The generic approach generates one predictive 
model for all pertinent patients, such as all patients 
admitted to intensive care units (ICUs). Eor example, 
the Acute Physiology and Chronic Health Evaluation 
(APACHE IV) and Simplified Acute Physiology Score 
(SAPS) are developed and applied to all patients 
admitted to ICUs.^ ^ A new^er generic model, the 
Laboratory-based Acute Physiology Score (LAPS), 
and the Comorbidity Point Score (COPS), w^ere 
developed for all patients admitted to acute-care hos- 
pitals,^ and used outpatient laboratory data from the 
24 h preceding the index hospitalization and diagno- 
sis data from the 12 months preceding the index hos- 
pitalization as predictor variables. This requires 
highly integrated inpatient and outpatient electronic 
systems, w^hich are currently less available than 
inpatient data only. Eurthermore, studies incorporat- 
ing classification of clinical conditions as covariates 
in the model tend to use proprietary classification 
systems, w^hich are not in the public domain. ^ 

The objectives of our study w^ere twofold: (1) use 
EHR data available at the time of inpatient admission 
to develop an Acute Laboratory Risk of Mortality 
Score (ALaRMS), to serve as a potential real-time 
decision support tool for patients admitted to acute 
care hospitals; and (2) add administrative data avail- 
able after patient discharge to the ALaRMS model to 
serve as a risk adjustment tool for retrospective 
inpatient outcome studies. To facilitate the standard- 
ization, reproducibility, and public access, w^e used 
the clinical classification algorithms of diagnoses in 
the public domain by the Agency for Healthcare 
Research and Quality (AHRQ).^^ 

METHODS 
Data 

We used one of the clinical research databases from 
CareEusion (San Diego, California, USA (formerly 
Cardinal Health/MediQual/MedisGroups)). This 
database has been used for research since the late 
1980s and the data collection system has been fully 
described elsew^here."^ ^ We used data from the 
EHR systems of 70 hospitals for all consecutively 
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hospitalized patients from 2006 through 2008. The laboratory 
data included numeric laboratory test results and collection times. 
A total of 23 numeric laboratory test results were included: 
serum chemistry (alphabetically ordered: albumin, aspartate trans- 
aminase, alkaline phosphatase, blood urea nitrogen (BUN), 
calcium, creatinine, glucose, potassium (K), sodium (Na), and 
total bilirubin); hematology and coagulation parameters (bands, 
hemoglobin, partial thromboplastin time, prothrombin time inter- 
national normalized ratio (PT INR), platelets, and white blood 
cell count (WBC)); arterial blood gas (partial pressure of carbon 
dioxide (PC02), partial pressure of oxygen (P02), and pH 
value); cardiac markers (brain natriuretic peptide (BNP), creatine 
phosphokinase MB (CPK MB), pro-BNI^ and troponin I). 

The database also included imported hospital administrative 
data comprising demographics, discharge disposition (which 
identifies inpatient mortality status), principal diagnosis, and up 
to 24 secondary diagnosis codes from the index hospitalization. 
We used discharges in 2006 and 2007 as the derivation cohort 
and discharges in 2008 as the validation cohort. A total of 95% 
of patients had laboratory data on the day of admission for both 
cohorts. For patients with multiple laboratory assessments on 
the admission day, we used the first reported value. 

Model development and validation 

Step 1: develop and validate ALaRMS 

We fit a logistic regression model using age, gender, and the first 
laboratory test results on the day of admission as candidate pre- 
dictor variables and inpatient mortality as the outcome variable. 
We partitioned age into 5 -year increments with age <30 years 
as the reference group. We partitioned each laboratory variable 
into multiple discretion levels based on the distribution of 
laboratory values and their associated mortality rates, and desig- 
nated the laboratory range associated with the lowest mortality 
rate as the reference group. We examined mortality rates for 
patients with missing values for each laboratory assessment and 
collapsed them into the corresponding reference group because 
their observed mortality rate was most comparable with that of 
the reference group. 

We converted the final laboratory model to an integer score 
system (ALaRMS) using a method described in the Framingham 
Study.^^ Specifically, we identified the variable with the smallest 
coefficient in the final multivariable model and applied it as the 
denominator. Then we divided each of the remaining regression 
coefficients in the model by this denominator and rounded the 
resulting quotient to the nearest whole number (integer), which 
formed the score weight for that variable. We then calculated 
each person's overall risk score by summing the points across all 
variables present. Converting model coefficients into a score 
system makes the risk adjustment model easy to understand and 
implement. We validated the ALaRMS using the validation 
cohort. 

Step 2: fit ALaRMS+CCS+CS model 

We summed all applicable laboratory points into a single 
ALaRMS value for each patient, and then fit a logistic regression 
model with the stepwise approach. We used the ALaRMS score 
as a continuous candidate variable along with candidate covari- 
ates of principal diagnosis-based clinical categories and secondary 
diagnosis-based comorbidity categories. We adopted AHRQ's 
clinical classification software (CCS)^^ and comorbidity software 
(CS)^^ as standard classification tools. The CCS collapses over 
14 000 ICD-9-CM diagnosis codes (International Classification 
of Diseases, 9th revision. Clinical Modification) into 285 



clinically meaningful categories. The CS grouped selected second- 
ary diagnosis codes into 30 comorbidity categories. 

Step 3: validate ALaRMS+CCS+CS model 

We vaHdated the ALaRMS + CCS + CS model using the vaHd- 
ation cohort. Specifically, we used the ALaRMS + CCS + CS 
model coefficients generated from the derivation cohort to 
score the validation cohort. We used the c-statistic and the 
Hosmer-Lemeshow statistic to evaluate the model fit. 

Analysis on the relative importance of ALaRMS versus CCS 
and CS variables 

We calculated the relative unique contributions of ALaRMS, 
CCS, and CS using methods described in previous research.^ ^ ^ 
Specifically, we calculated changes in the model fit log likelihood 
value when each group of variables was retained and then 
removed from the full model. 

To further evaluate the relative importance of ALaRMS versus 
CCS and CS variables in the mortality predictive models, we 
conducted the following additional analyses: (1) we fit four indi- 
vidual models using age and gender alone, laboratory variables 
alone, CCS alone, and CS alone, and compared the c-statistics; 

(2) we compared the cumulative change in the c-statistics by 
reversing the order of ALaRMS and the CCS and CS variables; 

(3) and we conducted integrated discrimination improvement 
(IDI) analysis, which assesses the new model's ability to improve 
the integrated sensitivity without sacrificing the integrated 
specificity.^^ 

Sensitivity analysis 

To test whether our model could be appHed to different types 
of patients, we conducted sensitivity analyses on model fit to 
different subgroups by patient age, medical versus surgical 
status, hospital teaching status, number of beds, and urban-rural 
status. 

All analyses were conducted using SAS V9.01. The study was 
approved by the New England Institutional Review Board/ 
Human Subjects Research Committee (Wellesley, Massachusetts, 
USA) and conducted in compliance with the Health Insurance 
Portability and Accountability Act (HIPAA), and the Helsinki 
Declaration. 

RESULTS 

Patient characteristics 

The derivation cohort comprised 1 428 824 discharges and 
34 147 deaths (2.4% mortality rate) (table 1). Median age was 
63 (IQR: 45, 78), and 58.9% were women. Approximately 
83.6% were white, 7.9% were black, and 8.5% were other race/ 
ethnicity. For payers, 37.7% were Medicare, 8.7% were 
Medicaid, and the rest (53.6%) were private or other payers. 
Approximately 64.1% of discharges were from teaching (n=36) 
and 35.9% were from non-teaching (n=34) hospitals; 52.6% of 
discharges were from small and medium (bed size <300) and 
47.4% were from large hospitals (bed size >300); 83.9% were 
from urban and 16.1% were from rural hospitals. The valid- 
ation cohort comprised 770 523 discharges and 18 456 deaths 
(2.4% mortality rate), with similar patient characteristics. 

ALaRMS model 

Age was a significant mortality predictor (table 2). Compared 
with patients aged less than 30 years (reference age group with 
OR of 1.00 and risk point weight of 0), patients aged between 30 
and 35 had a moderate increase in mortality risk (3 points), and 
patients aged between 35 and 39 old had a steep increase in 



456 



Tabak YP, et al.JAm Med Inform Assoc 2014;21:455-463. doi:1 0.1 136/amiajnl-201 3-001 790 



Research and applications 



Table 1 Patient characteristics by derivation and validation cohorts 


Variable 


Derivation cohort (2006-2007) 




Validation cohort (2008) 




Discharges, n (%) 


Mortality, n (%) 


Discharges, n (%) 


Mortality, n (%) 


Total 




^4 1 47 (1 4^ 


770 523 


1 R 4R6 D 4^ 

1 O H JU yZ .H-/ 












Age, median (IQR) 


63 (45, 78) 




63 (45, 78) 




Age <c65 


744 160 IS7 ^) 


7961 (1.1) 


40S S66 fS7 6) 


4444 (1.1) 


Age >65 


684 664 Idl ^) 


76 186 R 8^ 


^64 QS7 iAl A\ 


14 017 R 8^ 


Gender 










Men 


579 562 (41 .2) 


16 797 (2.9) 


318 966 (41.6) 


9282 (2.9) 


\A/nmon 

V VUI 1 ICl 1 


84Q 767 l^R 


1 7 ^RO (1 0^ 


4^1 RR7 C^R A) 


QI 74 D 0^ 


Raro/othnirit\/ 
liCILC/CLI M IILI Ly 










White 


1 177 QRQ IR"^ fs\ 

1 1 / / yOJ.U/ 


7R 774 D 4^ 

Z.O 1 1 '-t \£. .H/ 


600 ^OR 


1 4 740 D 


Black 


110 565 (7.9) 


2182 (2.0) 


63 908 (8.3) 


1191 (1.9) 


Other 


140 300 f8 5) 


3191 (2.3) 


106 310 ^^3 A) 


2525 (2.4) 












IVIcUILdic 


:J/17 ^7 7^ 


1 Q RR7 n 7^ 


771 ^47 4^ 


QQ47 ^ 7^ 


Morlirairl 

IVICUILCIIU 


177 6R0 19. 1\ 


1197 (1.0) 


70 1Q^ 7^ 


71 5 (1 .0) 


Pri\/ato nr nthor 
nivciLc: ui ULiici 


1 1 H Oi-I \JD.\Jj 


1 ^ ^QR n 7^ 


H-ZO 300 


77Q4 (^ 9) 

/ / Z7H y 1 .0/ 


Toarhinn ctatiic 
IcdLMIIiy oLdLUb 










Tparhinn 

1 CClV_l III 1^ 


qi i; :J7i f64 1^ 


77 004 0 4^ 


4Q6 4S7 f64 4) 


12 132 (2.4) 


IMUl 1 LcdLMIIiy 


R1 ^ RO^ 


17 14^ ^7 4^ 


774 071 6^ 


6^74 n 


Number of beds 










<300 


7R1 677 ^R7 6^ 


1 6 R68 D 7^ 

1 U OUO \£..£.] 


400 1 ^1 

H-VJVJ 1 J 1 \J\ .-7/ 


OJHD yZ . 1 / 


>300 


677 147 (47.4) 


17 279 (2.6) 


370 392 (48.1) 


9913 (2.7) 


Urban/rural status 










Urban 


1 198 334 (83.9) 


28 768 (2.4) 


655 528 (85.1) 


15 845 (2.4) 


Rural 


230 490 (16.1) 


5379 (2.3) 


114 995 (14.9) 


2611 (2.3) 


Medical/surgical discharges 










Medical 


939 081 (65.7) 


26 621 (2.8) 


514 932 (66.8) 


14 519 (2.8) 


Surgical 


489 743 (34.3) 


7526 (1.5) 


255 591 (33.2) 


3937 (1.5) 



mortality risk (10 points). Thereafter, the mortaHty risk increased 
2-4 additional points consecutively for every 5 -year increment. 

The laboratory covariates w^ith the highest mortality impact 
(risk scores >10; approximate ORs >2) w^ere albumin <2.4 g/dL, 
pro-BNP >18 000pg/mL, BUN > 55 mg/dL, arterial pH <7.2, 
arterial pH 7.21-7.30, arterial pH 7.31-7.35, P02 <50 mm Hg, 
P02 >140mmHg, bands >32%, platelets <115 000 cells/mm^ 
WBC >19 800cells/mm\ and troponin I >0.3 ng/mL or CPK 
MB >34 ng/mL (table 2). Detailed ALaRMS model parameters 
M^ith 95% CIs are presented in online supplementary appendix 
A. The ALaRMS model c-statistic w^as 0.87 w^ith good model cali- 
bration (figure lA). The predicted probability of mortality risk 
ranged from 0.004% to 99.3%. Mean ALaRMS score was 36 
(SD 21); median was 36 (IQR 22, 49). The results for the valid- 
ation cohort w^ere similar (figure lA). 

ALaRMS+CCS and CS model 

In addition to the ALaRMS variables, the logistic regression 
model yielded 35 CCS and 9 CS significant covariates (table 3). 
Every point increase in ALaRMS score resulted in an approxi- 
mately 6% increase in mortality risk (OR 1.057; 95% CI 1.056 
to 1.058). The principal diagnosis-based CCS variables w^ith 
ORs >2.00 included cardiovascular disease, major organ, hema- 
tologic, or metastatic cancers, brain or multiple traumas, and 
severe infectious diseases (eg, septicemia, HIV). Only one sec- 
ondary diagnosis-based comorbidity variable (metastatic cancer) 
in the CS had an OR >2.00. Adding CCS and CS variables 



increased the model c-statistic from 0.87 to 0.91. When the 
ALaRMS CCS and CS model coefficients w^ere appHed to the 
2008 vaHdation cohort, the model c-statistic w^as 0.90. The 
model exhibited good calibration for both the derivation and 
vaHdation cohorts (figure IB). 

Analysis on the relative importance of ALaRMS versus CCS 
and CS variables 

The relative contributions to the model predictive ability w^ere 
69% for ALaRMS, 25% for CCS, and 6% for CS. The c- 
statistics w^ere 0.704 for model w^ith age and gender variables 
alone, 0.843 for laboratory variables alone, 0.776 for CCS vari- 
ables alone, and 0.713 for CS variables alone. 

The c-statistic improved from 0.838 to 0.907 when ALaRMS 
was added to the CCS and CS model (table 4). In contrast, the 
c-statistic improved from 0.868 to 0.907 w^hen CCS and CS 
variables added to ALaRMS. 

The IDI analysis revealed that adding ALaRMS on top of 
CCS and CS variables improved the IDI by 127% (95% CI 
122% to 133%) (table 4). In contrast, adding CCS and CS vari- 
ables on top of ALaRMS only improved the IDI by 22% (95% 
CI 19% to 25%). 

Sensitivity analyses on model applicability to subgroups of 
patients 

The ALaRMS CCS -FCS model exhibited excellent discrimin- 
ation for age, teaching status, size (number of beds), location. 
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Table 2 Acute Laboratory Risk of Mortality Score (ALaRMS) 

Age 18-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 >89 



Gender 


Female 


Male 








Albumin (g/dL) 


<2.4 


2.5-2.7 


2.8-3 


3.1-3.3 


>3.3 


AST (U/L) 


<30 


31-40 


41-60 


61-100 


>100 


Total bilirubin (mg/dL) 


<1.4 


1.5-2 


>2.0 






Calcium (mg/dL) 


<7.9 


8-8.4 


8.5-10.1 


>10.1 




Creatinine (mg/dL) 


<2.0 


>2.0 








pro-BNP (pg/dL) 


<8000 


8001-18 000 


>18 000 






BNP 


<1200 


1201-2400 


>2400 






Glucose (mg/dL) 


<70 


71-135 


136-165 


>165 




K (mEq/L) 


<3.2 


3.3-4.9 


5-5.3 


>5.3 




Na (mEq/L) 


<130 


131-135 


136-143 


144-145 


>145 


Alkaline phos (U/L) 


<115 


116-220 


221-630 


>630 




BUN (mg/dL) 


<25 


26-30 


31-40 


41-55 


>55 


pH Arterial 


<7.2 


7.21-7.3 


7.31-7.35 


7.36-7.48 


>7.48 


P02 Arterial (mm Hg) 


<50 


50.1-55 


55.1-140 


>140 




pC02 Arterial (mm Hg) 


<35 


36-50 


>50 






PTT 


<22 


23-45 


45.1-55 


>55 




PT INR 


<1.1 


1.11-1.4 


1.41-2 


2.1-5 


>5 


Bands (%) 


<6 


7-13 


14-32 


>32 




Hemoglobin (g/dL) 


<10 


11-18 


>18 






Platelets (10^/L) 


<115 


115.1-150 


150.1-420 


>420 




WBC (1000/mm^) 


<4.3 


4.4-10.9 


11-14.1 


14.2-19.8 


>19.8 


Troponin 1 (ng/mL) 


<0.04 


0.05-0.1 


0.11-0.2 


0.21-0.3 


>0.3 


CPK MB (ng/mL)* 


<2 


3-5 


6-10 


11-34 


>34 





Reference range for each variable is presented in bold. 
*Use troponin I when available; otherwise use CPK MB. 

AST, aspartate transaminase; Alkaline phos, alkaline phosphatase; BNP, brain natriuretic peptide; BUN, blood urea nitrogen; CPK MB, creatine phosphokinase MB; PC02, partial 
pressure of carbon dioxide in arterial blood; P02, partial pressure of oxygen in arterial blood; pro-BNP, pro-brain natriuretic peptide; PTT, partial thromboplastin time; PT INR, 
prothrombin time international normalized ratio; WBC, white blood cell count. 



and medical versus surgical subgroups (table 5). The c-statistic 
for all subgroup analyses ranged from 0.86 to 0.94 and the caH- 
brations were good (figure 2). 

DISCUSSION 

Using over 2 million discharges from the EHR systems w^ith 
numeric laboratory results, w^e demonstrated that the laboratory 



results obtained first on the day of admission can generate a 
clinically valid mortality predictive model w^ith excellent predict- 
ive ability (the c-statistic of 0.87). ALaRMS may have utility for 
real-time decision support because it uses only EHR data available 
at the time of admission. The large sample size enabled precise par- 
ameter estimates as indicated by narrow^ CIs. We further demon- 
strated that AHRQ's standard principal diagnosis-based classification 
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Figure 1 Hosmer-Lemeshow 
calibration plot for: (A) the ALaRMS 
model; (B) the ALaRMS+CCS+CS 
model. ALaRMS, Acute Laboratory Risk 
of Mortality Score; CCS, clinical 
classification system; CS, comorbidity 
software. 




Expected Mortality Rate 




Expected Mortalit>' Rate 



(CCS) and secondary diagnosis-based comorbidity (CS) can be 
incorporated and further improve model c-statistic from 0.87 to 
0.91. The model that incorporated CCS and CS may be useful for 
retrospective outcome studies where post-discharge administrative 
data are available. 

Value of numeric laboratory data 

Laboratory results obtained at admission provide the data for 
objective assessment of patients w^ith acute clinical presentation. 
ALaRMS used only age, gender, and the initial laboratory test 
results, M^hich are most w^idely automated and commonly avail- 
able at the time of admission. This enables near real-time risk 
stratification that may be useful in aiding disease management. 
Patients who present v^ith severely deranged laboratory results 
have higher ALaRMS scores, indicating higher mortality risk. 
Conversely, patients w^ho present w^ithout abnormal laboratory 
results have low^er ALaRMS scores, indicating lovs^er mortality 
risk. Hence, ALaRMS can provide a near real-time aggregated 
assessment of severity in acute-care settings when discharge 
diagnosis data are not yet available. The potential utility in real- 
time settings needs to be validated in future prospective studies. 



For retrospective outcome studies and benchmarking, a valid 
risk adjustment model enhances clinical validity. The ALaRMS 
plus CCS and CS model minimizes potentially unfair risk adjust- 
ment due to up-, under-, or mis-coding. Since the ALaRMS 
score carries the largest proportion of the total w^eight in the 
risk adjustment model, w^hich is consistently demonstrated in c- 
statistic, log likelihood, and IDI analysis, it w^ould be desirable 
to incorporate it into the risk adjustment strategies. This w^ould 
represent another meaningful use of information technology in 
the healthcare settings. 

Laboratory data are quantitative, providing more precise and 
graded information on clinical severity. The graded relationship 
betw^een degree of abnormality of the laboratory measures and 
risk of mortality is not captured w^ith diagnosis code-based 
dichotomous variables, even w^hen these diagnosis codes are 
coded accurately. This explains in part the robust finding that the 
laboratory and physiology data contribute most to the predictive 
ability of the model in disease-specific and generic mortality pre- 
dictive models among inpatient and ICU patient populations.^^ ^ 

Laboratory results are parsimonious and potentially cost- 
efficient as relatively iew laboratory test results (23 in our 
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Table 3 


ALaRMS+CCS+CS model 








Variable 




Discharge, n (%) 


Unadjusted mortality. 


Multivariable adjusted 






n \ /o) i^i) 


Urx \VJ /O 

1 .UO ^ 1 .uO lU 1 .\JO) 


ALaRMS score, continuous variable (per 1 point increment) 


Clinical classification software (CCS) variables 








CCS100 


Acute myocardial infarction 


35 421 (2.5) 


2213 (6.2) 


2.58 (2.44 to 2.71) 


CCS102 


Non-specific chest pain 


39 691 (2.8) 


15 (0.0) 


0.07 (0.04 to 0.12) 


CCS103 


Pulmonary heart disease 


8818 (0.6) 


275 (3.1) 


1.93 (1.70 to 2.19) 


CCS108 


Congestive heart failure; non-hypertensive 


58 501 (4.1) 


1852 (3.2) 


1.58 (1.50 to 1.67) 


CCS109 


Acute cerebrovascular disease 


25 956 (1.8) 


2334 (9.0) 


8.41 (8.00 to 8.84) 


CCS114 


Peripheral and visceral atherosclerosis 


9632 (0.7) 


367 (3.8) 


2.63 (2.35 to 2.95) 


CCS115 


Aortic, peripheral, and visceral artery aneurysms 


5636 (0.4) 


476 (8.4) 


4.81 (4.32 to 5.34) 


CCS12 


Cancer of esophagus 


811 (0.1) 


72 (8.9) 


5.52 (4.26 to 7.16) 


CCS122 


Pneumonia (except that caused by tuberculosis 
or sexually transmitted disease) 


44 365 (3.1) 


1680 (3.8) 


1.64 (1.55 to 1.73) 


CCS129 


Aspiration pneumonitis; food/vomitus 


10 372 (0.7) 


1186 (11.4) 


3.15 (2.93 to 3.38) 


CCS13 


Cancer of stomach 


1277 (0.1) 


93 (7.3) 


3.47 (2.76 to 4.36) 


CCS131 


Respiratory failure; insufficiency; arrest (adult) 


17 462 (1.2) 


3140 (18.0) 


3.24 (3.08 to 3.41) 


CCS133 


Other lower respiratory disease 


5579 (0.4) 


217 (3.9) 


2.85 (2.47 to 3.29) 


CCS14 


Cancer of colon 


5723 (0.4) 


211 (3.7) 


2.03 (1.76 to 2.36) 


CCS145 


Intestinal obstruction without hernia 


15 040 (1.1) 


396 (2.6) 


1.98 (1.78 to 2.20) 


CCS159 


Urinary tract infections 


21 548 (1.5) 


217 (1.0) 


0.55 (0.48 to 0.63) 


CCS16 


Cancer of liver and intrahepatic bile duct 


966 (0.1) 


101 (10.5) 


2.99 (2.39 to 3.75) 


CCS17 


Cancer of pancreas 


1948 (0.1) 


164 (8.4) 


2.46 (2.07 to 2.93) 


CCS19 


Cancer of bronchus; lung 


7968 (0.6) 


849 (10.7) 


6.70 (6.19 to 7.26) 


CCS2 


Septicemia (except in labor) 


31 452 (2.2) 


4917 (15.6) 


2.82 (2.70 to 2.95) 


CCS203 


Osteoarthritis 


43 389 (3.0) 


42 (0.1) 


0.18 (0.13 to 0.24) 


CCS205 


Spondylosis; intervertebral disc disorders; other back problems 


35 984 (2.5) 


48 (0.1) 


0.27 (0.20 to 0.36) 


CCS233 


Intracranial injury 


8882 (0.6) 


703 (7.9) 


7.13 (6.53 to 7.78) 


CCS234 


Crushing injury or internal injury 


4062 (0.3) 


179 (4.4) 


2.93 (2.48 to 3.47) 


CCS249 


Shock 


133 (0.0) 


46 (34.6) 


6.11 (3.99 to 9.38) 


CCS27 


Cancer of ovary 


831 (0.1) 


53 (6.4) 


6.26 (4.65 to 8.42) 


CCS35 


Cancer of brain and nervous system 


1650 (0.1) 


61 (3.7) 


5.05 (3.88 to 6.57) 


CCS38 


Non-Hodgkin lymphoma 


2626 (0.2) 


215 (8.2) 


5.47 (4.70 to 6.37) 


CCS39 


Leukemia 


1784 (0.1) 


262 (14.7) 


6.45 (5.57 to 7.47) 


CCS40 


Multiple myeloma 


857 (0.1) 


73 (8.5) 


5.10(3.94 to 6.60) 


CCS42 


Secondary malignancies 


13 815 (1.0) 


1140 (8.3) 


4.97 (4.64 to 5.33) 


CCS43 


Malignant neoplasm without specification of site 


184 (0.0) 


31 (16.8) 


10.86 (7.1 to 16.6) 


CCS5 


HIV infection 


1255 (0.1) 


102 (8.1) 


6.44 (5.13 to 8.08) 


CCS50 


Diabetes mellitus with complications 


19 795 (1.4) 


172 (0.9) 


0.47 (0.41 to 0.55) 


CCS85 


Coma; stupor; and brain damage 


829 (0.1) 


87 (10.5) 


5.76 (4.43 to 7.48) 


Comorbidity software (CS) variables 










Congestive heart failure 


117 594 (8.2) 


9271 (7.9) 


1.48 (1.44 to 1.53) 




Depression 


150 995 (10.6) 


3018 (2.0) 


0.80 (0.77 to 0.83) 




Hypertension 


677 039 (47.4) 


18 104 (2.7) 


0.75 (0.74 to 0.77) 




Metastatic cancer 


33 342 (2.3) 


2751 (8.3) 


2.16(2.06 to 2.26) 




Other neurological disorders 


94 647 (6.6) 


4386 (4.6) 


1.37 (1.32 to 1.42) 




Pulmonary circulation disease 


25 575 (1.8) 


2234 (8.7) 


1.46 (1.39 to 1.54) 




Renal failure 


135 725 (9.5) 


8127 (6.0) 


1.18 (1.14 to 1.22) 




Solid tumor without metastasis 


28 198 (2.0) 


1506 (5.3) 


1.51 (1.43 to 1.61) 




Weight loss 


35 984 (2.5) 


3836 (10.7) 


1.56 (1.50 to 1.63) 


ALaRMS, Acute Laboratory Risk of Mortality Score. 

• 



ALaRMS model) are needed to assess the major organ/system 
functions that keep patients alive. The objective, quantitative, and 
parsimonious nature of numeric laboratory data exhibited high 
predictive ability as show^n in the ALaRMS model. Although not 
directly comparable, the ALaRMS model's c-statistic of 0.87 
represents higher model discrimination than the c-statistics of 
0.70, 0.71, and 0.72 achieved by the congestive heart failure, 
acute myocardial infarction, and pneumonia models derived from 
administrative claims that are currently used by the Centers for 



Medicare and Medicaid Services (CMS) in the USA for the 
Hospital Compare w^ebsite.^^"^^ Nevertheless, it should be 
emphasized that our model predicted inpatient mortality and the 
CMS models predicted 30-day mortality, hence the absolute c- 
statistic is not directly comparable. Our full model (ALaRMS 
+ CCS and CS) achieved a c-statistic of 0.91, w^hich is slightly 
higher than the c-statistic of 0.88 previously reported in a risk 
adjustment model using pre-admission laboratory and administra- 
tive data for inpatient population.^ 
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Table 4 Integrated discrimination improvement (IDI) 

IDI 

Died in hospital (n=34 147) Live discharge (n=1 394 677) 

Integrated Standard Integrated Standard Discrimination 



Model 


c-Statistic 


sensitivity (IS) 


deviation (IS) 


1 -specificity (IP) 


deviation (IP) 


slope 


IDI (95% CI) 


Adding ALaRMS to CCS and CS 
















CCS and CS 


0.838 


0.0910 


0.0974 


0.0223 


0.0398 


0.0688 




dllU V_ j-rMLdr\IVI J 


0 907 


0 1 766 


0 1 97 


0 0202 


0 051 5 


0 1 564 




Difference 




U.Uojj 


u. 1 / jy 


— U.UUz 1 




U.Uo/D 


U.Uo/D ^U.Uojo 
to 0.0895) 


% Improvement by adding 




94% 




9% 




127% 




ALaRMS (95% CI) 












(122%, 133%) 




Adding CCS and CS to ALaRMS 
















ALaRMS 


0.868 


0.1496 


0.1894 


0.0208 


0.046 


0.1287 




ALaRMS+CCS and CS 


0.907 


0.1766 


0.197 


0.0202 


0.0515 


0.1564 




Difference 




0.027 


0.0944 


-0.0007 


0.0276 


0.0277 


0.0277 (0.0267 
to 0.0287) 


% Improvement by adding 




18% 




3% 




22% 




CCS and CS (95% CI) 












(19%, 25%) 





Calculation of % of improvement: 

Adding ALaRMS to CCS&CS: 

DSccs&cs=ISccs&cs-IPccs&cs=0.0910-0.0223=0.0688 

DSccS&CS+ALaRMS=IScCS&CS+ALaRMS-IPcCS&CS+ALaRMS=0-1 766-0.0202=0.1 564 

IDI of adding ALaRMS=DSccs&cs+ALaRMs-DSccs&cs=0.1 564-0.0688=0.0876 
% of improvement by adding ALaRMS=IDI/DSccs&cs=0.0876/0.0688=127% 
Adding CCS&CS to ALaRMS: 

DSALaRMS=ISALaRMS-IPALaRMS=0.1 496-0.0208=0.1 287 

DSALaRMS+CCS&CS=ISALaRMS+CCS&CS-IPALaRIVlS+CCS&CS=0. 1766-0.0202=0.1 564 

IDI of adding CCS&CS=DSALaRMs+ccs&cs-DSALaRMs=0.1 564-0.1 287=0.0277 
% improvement of adding CCS&CS=IDI/DSALaRMs=0.0277/0.1 287=22% 

ALaRMS, Acute Laboratory Risk of Mortality Score; CCS, clinical classification system; CS, comorbidity software; IS, integrated sensitivity=mean predicted probability of mortality in the 
group of patients died in hospital; IP, integrated 1-specificity=mean predicted probability of mortality in the group of patients discharged live; Discrimination slope, IS-IP; IDI, 
integrated discrimination improvement=(IS [new]-IS [old])-(IP [newj-IP [old]). 



Value of using standard clinical classification system in the 
public domain 

Adopting a standard clinical condition classification system is 
critical because it allows for public access, enables future valida- 
tions using different datasets, and offers opportunities to further 
improve the risk adjustment model through refinement of the 
laboratory score algorithms and code-based clinical condition 
classification system. The AHRQ's CCS has undergone multiple 



revisions and is updated annually to reflect ICD-9-CM 
updates/^ and is perhaps the most feasible clinical classification 
system available to the public. 

Principal diagnosis-based groups are especially important for 
a generic risk adjustment model because general patient popula- 
tions are heterogeneous. As in our model, patients with princi- 
pal diagnoses (primary reason for the index hospitalization) of 
major acute clinical conditions of vital organs/systems (eg. 



Table 5 Sensitivity analysis: model discrimination for subgroups 



Variable 



Derivation cohort 



Number of discharges 



Model 
c-statistic 



Validation cohort 



Number of discharges 



Model 
c-statistic 



Total 1 428 824 

Age 

Age<65 744 160 

Age >65 684 664 

Teaching status 

Teaching 915 321 

Non-teaching 513 503 

Number of beds 

<300 751 677 

>300 677 147 

Urban/rural status 
Urban 1 198 334 

Rural 230 490 

Medical/surgical discharges 

Medical 939 081 

Surgical 489 743 



0.907 

0.939 
0.863 

0.903 
0.910 

0.909 
0.907 

0.908 
0.905 

0.903 
0.910 



770 523 

405 566 
364 957 

496 452 
274 071 

400 131 
370 392 

655 528 
114 995 

514 932 
255 591 



0.903 

0.935 
0.856 

0.900 
0.904 

0.903 
0.903 

0.903 
0.900 

0.897 
0.910 
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Expected Mortality Rate 

Figure 2 Hosmer-Lemeshow calibration plot for subgroup patients: (A) age 65 or older versus age younger than 65; (B) discharges from teaching 
versus non-teaching hospitals; (C) medical versus surgical discharges; (D) discharges from large (>300 beds) versus small/medium-sized (<300 beds) 
hospitals; (E) discharges from urban versus rural hospitals. 



cardiovascular or neurologic systems, cancers of major organs/ 
systems, dissimilated cancers) have high mortality risk. In con- 
trast, clinical conditions involving less-vital organs/systems (eg, 
joint and muscular system) or less-acute diagnosis (eg, unspecific 
chest pain) have low^er mortality risk. These results are valid 
clinically and are consistent w^ith the mortality statistics reported 
by the National Center for Health Statistics.^^ 

Among secondary diagnosis-based comorbidity variables, 
metastatic cancer carried the highest independent risk of mortality. 
Other comorbid conditions tended to be either statistically insignifi- 
cant or carry less independent w^eight because they are likely 
accounted for by ALaRMS physiology parameters. The 
less-prominent contributions of secondary diagnosis-based comorbid 
conditions are consistent w^ith previous risk adjustment models that 
incorporated physiology data.^^ ^ 

Limitations 

Our study has limitations. First, although our dataset comprised 
over 2 million discharges from 70 hospitals from the northeast 



region, it is not geographically representative of the US patient 
population. Further vaHdation of this model is needed w^hen 
data from a more representative patient population become 
available in the future. Second, w^e did not include vital sign and 
altered mental status data in the physiology risk score because 
our goal was to develop a model using only data that are cur- 
rently w^idely available in hospital EHR systems. Inclusion of 
vital signs and mental status requires incorporating nursing 
notes or other more complex electronic medical records 
systems, w^hich are not as w^idely available as numeric laboratory 
data in acute-care hospital settings. At the time when vital sign 
and mental status data are w^idely captured electronically across 
hospitals, it w^ould be important to incorporate them into the 
physiology score because of their clinical face validity. Third, w^e 
did not incorporate AHRQ's procedure groups in our model 
because many of the procedures are diagnostic in nature. For 
procedure groups indicating treatment, it is not possible to dis- 
tinguish w^hether a treatment procedure w^as implemented early 
at admission as planned or at a later hospital stay after patients 
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possibly developed complications. The latter should not be 
included as a risk adjustor because it would give credit (higher 
expected mortality rates) to hospitals w^ith potentially substand- 
ard care if they have more potential complications requiring 
procedural treatments after admission. On the other hand, omit- 
ting procedure categories is unlikely to have a large impact on 
the model predictive ability because planned treatment proce- 
dures w^ould likely be correlated w^ith principal diagnosis cat- 
egories. For example, patients receiving cardiac operations 
M^ould likely have cardiac disease diagnoses, w^hich have already 
been included in the model. Their acute clinical presentation 
and severity w^ould have been largely taken into account w^ith 
ALaRMS. As further evidence, the high c-statistic of our model 
(ranged from 0.90 to 0.91, nearly identical for medical and sur- 
gical patients) indicated excellent predictive ability for both 
medical and surgical patients in our sensitivity analysis. 

Significance of the study 

The data derived from an EHR typically covers many broad 
domains. Numeric laboratory data are objective and quantitative 
in nature, w^hich are desirable features in predictive modeling. 
The objective nature ensures the data reliability and reproduci- 
bility. The quantitative nature enables precise estimation of the 
graded relationship of the degree of physiologic derangement 
and the risk of inpatient mortality. Furthermore, numeric 
laboratory data are perhaps most scalable among all EHR 
domains due to the objective and quantitative nature. While the 
health informatics field has devoted considerable effort to 
extract data from free text reports or expensive but infrequently 
used tests, it is w^orthw^hile to examine the utility of the most 
commonly measured, most scalable, and perhaps the least 
expensive data domains. This is particularly true in regard to 
real-time decision support and outcome studies, w^hich often 
require a large scale implementation at a low^ cost. 

CONCLUSIONS 

Admission laboratory physiologic data captured in the hospital 
EHR systems provide objective, precise, and parsimonious 
assessment of acute clinical severity and are highly predictive of 
inpatient mortality risk in hospitalized adult patients. Relying 
only on data available at the time of admission, ALaRMS may 
have utility in aiding real-time disease management. 
Incorporating AHRQ's standardized clinical classification 
systems (CCS and CS) further improves model predictive power 
and facilitates public access to the risk adjustment algorithms. 
Using a completely automated EHR dataset available on admis- 
sion, ALaRMS can be implemented for real-time decision 
support. The full model incorporating CCS and CS may be cost- 
efficient to implement for large-scale retrospective studies on 
inpatient outcomes. These w^ould represent meaningful use of 
health information technology. 
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